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The NASA developed Human Research Facility 1 (HRF1) and Human Research Facility 
(HRF2) experiment racks have been operating in the European Space Agency (ESA) 
Columbus module of the International Space Station (ISS) since Summer 2008. The two 
racks are of the same design. Since the start of operations, unexpected pressure spikes were 
observed in the Columbus module’s thermal-hydraulic system during the racks activation 
sequence. The root cause of these spikes was identified in the activation command sequence 
in the Rack Interface Controller (RIC), which controls the flow of thermal-hydraulic system 
fluid through the rack. A new Common RIC Software (CRS) release fixed the bug and was 
uploaded on both racks in late 2009. This paper gives a short introduction to the topic, 
describes the Columbus module countermeasures to mitigate the spikes, describes the 
ground validation test of the new software, and describes the flight checks performed before 
and after the final upload. Finally, the new on-orbit test designed to further simplify the 
racks hydraulic management is presented. 
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Nomenclature 


ATCS 

CRS 

ESA 

FGSE 

ISS 

MSFC 

PRCU 

RFCA 

RIC 

SFCA 


Active Thermal Control System 
Common RIC Software 
European Space Agency 
Fluidic Ground Support Equipment 
International Space Station 
Marshall Space Flight Center 
Payload Rack Checkout Unit 
Rack Flow Control Assembly 
Rack Interface Controller 
System Flow Control Assembly 


I. Introduction 


A number of NASA facilities are designed around 
the concept of providing a set of common interfaces 
for small, modular experiments. This allows research 
objectives to change over time and for the common 
packaging of experiments. This is the concept behind 
the NASA EXPRESS racks. Several middeck and 2 
SIR drawer slots are available in each EXPRESS 
rack. There are currently eight EXPRESS racks on- 
orbit in the US Laboratory, the European Columbus 
module and the Japanese Experiment Module. 

The design of the EXPRESS rack was extended to 
a few other research facilities; the Window 
Observation Research Facility (WORF), the Habitat 
Holding Racks (HHR), and the two HRF racks. The 
HHR was part of the Centrifuge Facility planned for 
the ISS, but the project was cancelled and the 
Centrifuge Facility was never completed. These other 
facilities are sometimes referred to as “EXPRESS 
derivatives”. 



T HE two Human Research Facilities HRF1 and HRF2 are two NASA experiment racks operating in Columbus, 
the ESA laboratory on the ISS since Summer 2008. The HRF1 one rack was the first research facility on the ISS 
and has been operating successfully on-orbit since March of 2001. HRF2 was delivered to orbit on a later flight and 
both racks were transferred to the Columbus module shortly after the module was commissioned on-orbit. 

The design and science objectives of these two racks are discussed in detail in two previous papers [1, 2]. It is 
summarized here again. 


The HHR and EXPRESS racks share common 
design features. The thermal control system in each 
rack has three internal flow controllers that use a 
modulated solenoid valve, flow meter and a dedicated 
PID controller. The RIC controls the flow of water by Figure 1. EXPRESS Rack 8 

providing a voltage to the flow controller and the flow meter provides a signal back to the RIC which is downlinked 
in the rack’s Health and Status data. One of the active flow controllers is a normally open valve and controls the 
flow of cooling water to the rack subsystems. The other two controllers are normally closed valves that control the 
flow of cooling water to two legs of the flow system supporting the experiments. 


The common design features of the HRF and EXPRESS racks mean that the RIC software for all of the 
EXPRESS and EXPRESS derivative is also common. The Common RIC Software (CRS) is sustained by the 
EXPRESS Program at Marshall Space Flight Center and is developed and tested in the software and hardware test 
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facilities there. The EXPRESS Rack 3 is also located in the Columbus module and shares the same CRS as the HRF 
racks. 

ESA and NASA have an extensive history of operation of the HRF racks in Columbus that began with ground 
testing of a HRF prototype rack in the Columbus module in Bremen. The results of this ground testing led to the 
implementation of start-up and shut-down procedures to minimize potential of a water hammer in the Columbus 
ATCS that can occur if the solenoid valve is allowed to suddenly close. The RIC software also contains an algorithm 
to slow the transition from one flow rate to another. After operations of the HRF racks and EXPRESS Rack 3 began 
in Columbus, the Columbus operations team noticed that during startup and shutdown of the HRF racks there were 
pressure spike noted in the Columbus ATCS. An Anomaly Resolution Team (ART) was convened with participation 
from both ESA and NASA teams to determine the cause of the pressure spikes and to find a resolution. 

The HRF racks were allowed to continue science operations during this time, but special steps were put in place 
during the startup of the racks. The WFSV for each rack was modulated to miminize the flow and pressure 
differential across the rack’s ATCS interface to minimize the pressure spikes. The pressure spikes on shutdown were 
resolved by changing the shutdown procedure for the racks. 

The ART determined that the pressure spikes during startup were caused by the power on sequence of 
components in the racks. The CRS at the time was powering the system flow control valve, and then sending a 
signal to the valve to be in the normally open position. This allowed the initial flow rate signal from the RIC to be in 
an initial intermediate state, and the valve would move to an initial intermediate position. It would then receive the 
signal to open fully. These two transitions occurred rapidly and caused the pressure spike in the system. 

The CRS was modified to first send the flow rate signal to command the valve fully open, and then power on the 
valve and flow controller. Ground tests were then conducted to ensure the desired result was obtained prior to 
updating the on-orbit software. The ground test results and on-orbit results are described below. 


II. CRS5 to CRS6 Ground Test Validation 

The new CRS release (CRS6) with the bug fix was designed and verified at Software Test Bed at Boeing 
premises on mid 2009 and tested at rack level using the Express Flight Rack 8 (Figure 1) at NASA-MSFC. The 
major test objective was to verify the absence of pressure spike during rack activation and deactivation when the 
thermal flow control algorithm is not active. Same set-up was run with CRS5 to further confirm the control 
algorithm improvement. 


A. Test Set-Up 

The FGSE adopted for the ER8-CRS6 test was the Payload Rack Checkout Unit (PRCU) the functional 
hydraulic layout of the test set-up is shown in Figure 2. 
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PRCU provides ISS I/F simulation and was already used by NASA to verify interface requirements of 
pressurized payloads to be installed on board ISS 3 . In particular, the PRCU is a ground simulator of the US Lab 
Internal Temperature Control System (ITCS) and is therefore different to the Columbus Active Thermal Control 
System (ATCS) for the following reasons: 


• PRCU pressure drop control works with a System Flow Control Assembly (SFCA) subsystem that controls the 
plenum pressure drop by changing the stroke of the plenum by-pass, i.e., the SFCA internal control valve (CV 
in Figure 2). The PRCU pump works at constant speed. 

Columbus ATCS does not have any equivalent SFCA assembly and the plenum pressure drop is guided by 
changing the ATCS pump speed 


• The SFCA set point of nominally 13 PSI, is more than twice the Columbus set-point of 6. 1 PSI 


• The Rack flow Control Assembly (RFCA) has an internal active control loop, while the equivalent Columbus 
valve, the Water Flow Selection valve (WFSV) does not. For this test (and on orbit in the US Lab), the RFCA 
was not in active control mode when the rack was working: the RFCA valve was set prior to ER8 activation to a 
certain flow rate and then blocked to that position with feedback control deactivated. 

For this test ER8 was equipped with two flex hoses, thus enabling thus the flow through the two subsystem legs 
and the intervention of relevant hydraulic control systems. Flow and pressure drop measurements from SFCA, 
RFCA and internal flow meters were a sampled with a frequency of 1Hz. Additional pressure sensors PI and P2 that 
measured the supply and return pressure respectively (Figure 2), were introduced to catch spike events; data were 
acquired at a frequency of 100 Hz. 
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All tests performed are summarized in the Table 1 


Case 

CRS 

Date 

SFCA 

RFCA 

Note 

(PSI) 

(kPa) 

(Lbs/h) 

(Kg/h) 

- 

5 

15-0ct-2009 

13 

89.6 

225 

102 

US Lab AP 

A 

6 

20-0ct-2009 

13 

89.6 

225 

102 

US Lab AP 

B 

6 

20-0ct-2009 

13 

89.6 

225 

102 

US Lab AP 

C 

6 

20-0ct-2009 

13 

89.6 

157 

71 

US Lab AP 

D 

6 

20-0ct-2009 

13 

89.6 

157 

71 

US Lab AP 

A1 

6 

6-Nov-2009 

6.1 

42 

157 

71 

Columbus AP 

B1 

6 

6-Nov-2009 

6.1 

42 

220 

100 

Columbus AP 

Cl 

6 

6-Nov-2009 

6.1 

42 

220 

100 

Power removal 


Table 1. ER8-PRCU Tests Cases 


B. CRS5 Test Results 


P1&P2; CRS5 Test Case (SFCA @ 13 PSI); 15-0ct-2009 



time (hh:mm) 


P1-P2 



First test was performed with 
CRS5 software to confirm the 
presence of spikes during activation 
for the hardware configuration 
selected. Test was configured with 
SFCA at 13 PSI (~90 kPa) and 
RFCA set to provide an initial flow 
(i.e., with ER8 unpowered) of about 
225 lbs/hr (-102 Kg/h). 

In Figure 3 are shown the 
Dewetron P1&P2 data (first plot) 
and their difference (second plot), 
from rack power on until 
completion of the start-up sequence. 
The activation spike at 25.7 kPa is 
clearly visible, thus confirming the 
validity of the test set-up adopted. 
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C. CRS6 Test Results with SFCA at 13 PSI 


P1&P2; CRS6 Test Case A (SFCA @ 13 PSI); 20-0ct-2009 
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Figure 4. ER8 test data with CRS6 and SFCA at 13 PSI (Case A) 


13:30 


First 4 tests with new CRS6 
were performed with SFCA set 
as working in the US Lab, i.e., 
with a set point of 13 PSI. 

The first two tests (cases 
A&B in Table 1) had the RFCA 
configured to provide 225 lbs/hr 
(102 Kg/hr) to ER8, followed 
then by two other tests (cases 
C&D in Table 1) with RFCA 
configured to provide 157 lbs/hr 
(71 Kg/hr) to ER8. 

For all cases the rack started 
from power-off position with 
avionic leg fully open and the 
sub-systems legs fully closed. 
Once the rack was turned on, it 
went through the start-up 
sequence, with the avionic valve 
from fully open to the flow set- 
point, approximately 45 kg/hr. 

The P/Ls legs were not 
commanded open at any point 
during these 4 flow testing. 
Before rack shut down, the 
avionic valve was commanded 
back to fully open position as for 
current flight operations. 

Figure 4 shows four plots of 
the test case A, comprising 
(starting from top) as follows: 
pressure sensors reading PI and 
P2 (1 st plot), rack pressure drop 
P1-P2 (2 nd plot), RFCA DP3 AP 
measurement (3 ld plot) and 
RFCA FM3 flow rate 
measurement (4 lh plot). 


All plots cover the activation and deactivation of the rack. No ER8 (internal flow) and SFCA data were retrieved 
for these cases. 

For this case, as for the other three cases tested at 13 PSI, no activation/deactivation spikes were measured: first 
CRS6 test day was thus performed successfully, giving the go for next tests to be performed. 
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D. CRS6 Test Results with SFCA at 6.1 PSI 


P1-P2 raw data; CRS6 Test Cases B&C (SFCA @ 6.1 PSI); 6-Now2009 



P1-P2 filtered (blue) + SFCA DP (red) + RFCA DP3(cyan) 



ER8 FMs [Avio (red) + Legl (black) + Leg2 (green) + overall (blue)] + RFCA FM3 (cyan) 


Additional tests were 
performed few weeks later, with 
SFCA set point reduced from 13 
PSI to 6. 1 PSI, as applicable for 
the Columbus plenum pressure 
drop, and in particular: 

• Start up and shutdown with 

RFCA configured to 
provide 157 lbs/hr (71 
Kg/hr) to ER8. This case is 
to be compared with cases 
C&D performed at 13 PSI, 
for which the only 
difference is in the SFCA 
set-point (case A1 in Table 
1 ) 

• Start up with RFCA full 

open. In this configuration 
with the rack off, the flow 
through the rack was 
approximately 220 lbs/hr 
(100 Kg/hr). This case was 
selected in order to have a 
worst case condition, i.e., 
simulating the WFSV fully 
open condition (case B1 in 
Table 1) 



» After completion of start up 
in test Bl, the rack was 
configured directly for full 
flow by first setting the 
subsystem/avionics valve to 
180 lbs/hr (82 Kg/hr), then 
setting payload leg 1 to 100 
lbs/hr (45 Kg/hr), and 
finally setting payload leg 2 
to 100 lbs/hr (45 Kg/hr) for 
a total flow through the rack 
of approximately 380 lbs/hr 
(172 Kg/hr). At this 

configuration power was then removed from the rack to record the water spike/hammer event (case Cl on Table 
1). Aim of last test was not to verify the CRS6 itself, but to measure the difference between the SFCA pressure 
spike, corresponding to the telemetry pressure drop data and the pressure peak measured by P1&P2 sensors, 
acquired with higher sampling rate. This data was useful to extrapolate the real pressure stress acting on 
Columbus Delta Pressure Sensor Block (DPSB) when EXPRESS racks power removal occurs. 


time (hh:mm) 

Figure 5. ER8 test data with CRS6 and SFCA at 6.1 PSI (Cases B+C) 


For this test day also SFCA and ER8 data were retrieved. In Figure 5 are reported three plots of the test cases Bl 
and Cl, composed (starting from top) as follow: AP =P1-P2 raw data as acquired with a sampling rate of 100 Hz 
(1st plot), AP filtered (blue line) to reduce noise and compared with RFCA DP3 AP (cyan line) and SFCA DP (red 
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line) measurements, acquired with a sampling rate of 1 Hz (2nd plot) and RFCA FM3 flow rate measurement (cyan 
line) vs ER8 legs and overall flow rates (3rd plot), with a sampling rate of 1 Hz. 

Major outcome is that for the nominal activations, no spike were observed also on this test day, showing that 
CRS6 fix was properly designed and implemented and no impacts expected depending on the working 
SFCA/plenum AP set-point. Based on this result, the go for CRS6 S/W upload in orbit was given. 

III. Flight Telemetry Data 

After successful completion of ground tests, the CRS6 was uploaded on-orbit on HRF1 and HRF2 in late 
October 2009. In the following flight telemetry of last activation with CRS5 and first activation with CRS6 for the 
HRF1 are shown to highlight software improvement benefits. For each activation three plots provide overview and 
details on activation, showing the most relevant data, both from Columbus and HRFs side, and in particular: 

Plot 1. WFSV position (opening %) 

Plot 2. WPA Flow rate (Kg/h) - this value is filtered to reduce noise 
Plot 3. DPSBAP(kPa) 

Plot 4. HRF partial and overall flow rates 


E. DOY 300/2009 HRF1 Activation with CRS5 


COL GMT 300-09 WFSV 



Last HRF 1 activation with 
CRS5 installed was performed at 
DOY 300/2009. Columbus 
water loop configuration was 
with initial pump flow rate of 
about 370 kg/h, due to MSG 
rack. WFSV was fully opened 
bringing the pump flow rate up 
to 420 kg/h and then partially 
closed down to about 28.5%. 

Pressure spikes due to 
WSFV opening/closure were 
38.6 kPa and 44.6 kPa 
respectively. HRF1 activation 
produced a small spike of 41.1 
kPa, due to CRS5 S/W still in 
place. This spike was within the 
control dead band thanks to 
graceful activation sequence 
which was still in place. 


HRF1 GMT 300-09 
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F. DOY 306/2009 HRF1 Activation with CRS6 


COL GMT 306-09 WFSV 



COL GMT 306-09 WPA Flow 



The first activation of HRF 1 
with the CRS6 installed was 
performed on DOY 306/2009 

A for DOY300, WFSV pre- 
positioning for rack activation 
produced spikes from 38.6 kPa 
to 44.4 kPa, while with respect 
GMT300 no pressure spike was 
measured at DPSB level during 
F1RF 1 activation, confirming the 
expected behavior of the rack 
with the new control S/W. 

After activation the WFSV 
stroke was adjusted to 51.8% 
and F1RF reconfigured 
producing a low peak at 38.7 
kPa. The rack overall flow of 86 
kg/h was in line with expected 
figure. 

Same comparison was 
performed on HRF2 activations, 
showing the same behavior of 
HRF1. 

Nominal activation sequence 
for the F1RF racks was then 
restored after about one year, 
with WFSV valves opening 
increased to 32% and 38% 
respectively for F1RF1 and 
HRF2. 


IV. Next HRF On-Orbit Tests 

Each F1RF P/L has five different working configurations, with corresponding valves settings, named Thermal 
Cases labeled from A to E, where A is the start-up configuration and E the most demanding flow. 

Before any transition between Thermal Cases, the stroke of the upstream Columbus WFSV must be set to a pre- 
defined position, depending on Thermal Case selected. WFSV setting was originally conceived to provide a more 
stable configuration for rack internal valves, as derived from ground compatibility test 1 , limiting also the flow rate 
through the rack in case of rack transient or failure when all HRF valves are in fully open configuration. 

After CRS6 upgrade, it is proposed to verify on-orbit the real need of intermediate WFSV positions, by testing 
the dynamic of HRF1 and HRF2 working with incremental WFSVs openings, up to an enveloping position that can 
allow an independent management of the Thermal Cases from P/Ls side but at the same time compliant with the 
rack flow rate limitation of 190 kg/h. Other advantages of having only one WFSV opening is the reduction of 
pressure spikes due to WFSV movements itself. 
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V. Conclusion 

The Common RIC Software, version CRS5, controlling the rack activation sequence in the HRF1 and HRF2 
operating in the European Space Agency (ESA) Columbus module since early 2008, was identified as the cause of 
unexpected pressure spikes observed in the thermal-hydraulic system. An upgraded CRS6 software version, which 
changed the rack activation sequence, was first successfully ground tested in a Software Test Bed facility and then in 
the Payload Rack Checkout Unit (PRCU) at NASA-MSFC using the Express Flight Rack 8 as test rack. No 
unexpected pressure spikes were observed for both the PRCU Rack Flow Control Assembly (RFCA) set point of 13 
PSI as in the US Lab, or 6.1 PSI for the applicable plenum pressure drop in Columbus (even with worst case 
condition of RFCA fully open prior to rack activation to simulate Columbus Water Flow Selection Valve (WFSV) 
fully open). 

The CRS6 software was uploaded to the HRF1 and HRF2 racks in October 2009. Subsequent in-orbit testing for 
HRF1 activation produced pressure spikes of 38.6 kPa and 44.4.kPa: no pressure spike was observed at the Delta 
Pressure Sensor Block level confirming the expected behavior of the CRS6 rack control software. Subsequent 
adjustment of the relevant WFSV to 51.8% open, produced only a low peak of 38.7 kPa. Similar behavior was 
observed for HRF2. 

The successful on-orbit testing of the CRS6 rack control software led to nominal rack activation being restored 
with WFSV valve opening set to 32% and 38% for F1RF1 and F1RF2 respectively. Future testing is proposed to 
identify an enveloping setting of the WFSV that allows independent control of the five HRF thermal cases while 
concurrently limiting the rack flow rate to the maximum allowed 190 kg/h. Use of a unique WFSV setting for all 
rack thermal cases will also reduce pressure spikes induced by the WFSV movement. 

The investigation, countermeasure development and implementation, and resolution of the delta pressure sensor 
spike problem detailed in this paper, was a milestone moment for Integrated Space Station Operations, since this 
was the first time ESA and NASA Engineering and Operations teams have worked so closely together to resolve an 
in-orbit anomaly impacting both the ESA Columbus Module and NASA Payload Operations. 

Through the close and successful cooperation of the various European and American teams it was possible to 
continue operations via an agreed countermeasure (workaround) strategy at an early stage of the problem 
investigation and also overcome the technical challenges of using a space station thermal loop test set up not 
originally designed to represent Columbus, to demonstrate effectively the software solution to the problem before 
uplinking and implementing it in-orbit. 

What is more the cooperative work that has been put into addressing the anomaly has fostered a positive 
environment which has lead not only to the resolution of the original problem but also the commitment to optimize 
F1RF Payload Operations in the future, through proposed further in-orbit testing. 

It is hoped by the authors, that through the experience that has been gained by ESA and NASA in working 
together to resolve the delta pressure spike anomaly (the details of which are reflected in this paper), a good working 
basis has been achieved for facing future technical challenges in the effort to support continued successful NASA 
Payload Operations in Columbus. 
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